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Q ' We report on a search for charmless hadronic B decays to the three- 

body final states K$h + ir~, K + h~ir°, Kgh + -ir° (h^ denotes a charged pion 

^ . or kaon), and their charge conjugates, using 13.5 fb _1 of integrated lumi- 

nosity produced near \fs = 10.6 GeV, and collected with the CLEO de- 
\ tector. We observe the decay B — > K°ir + ir~ with a branching fraction 

(50j^°(stat.)±7(syst.)) x 10" 6 and the decay B -> K* + (892)ir- with a branch- 

^ ■ ing fraction (16^(stat.)±2(syst.)) x 10~ 6 
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Recent years have seen [1] the first observations of several two-body charmless hadronic 
decays of B mesons, including the four B — > Kn transitions. These two-pseudoscalar decays 
have received considerable attention due to their expected role in improving our understand- 
ing of the weak interaction and in the extraction of the complex quark couplings described 
by the Cabibbo-Kobayashi-Maskawa matrix [2]. The pseudoscalar- vector analogs of these 
decays, B — > K*n and Kp, provide further constraints on the magnitudes and phases of 
these couplings and present additional opportunities for observing direct CP violation [3] 
and deviations from the Standard Model. Accurate measurement and interpretation of the 
rates and CP asymmetries of these quasi-two-body decays requires knowledge of the non- 
resonant contributions to the same final states [4]. In this Letter, we report on a study of B 
decays to the three-pseudoscalar final states, Kgh + ir~, K + h~7r°, and Kgh + n° (/i ± denotes a 
charged pion or kaon), without regard for the resonant substructure. For each final state, we 
also search for two-body channels with intermediate vector resonances, as well as for non- 
resonant production. Results for the Kgh + n^ and K + h~n topologies have been presented 
previously by the CLEO [5] and Belle [6] Collaborations. The inclusion of charge conjugate 
states is always implied. 

The data sample used in this analysis was produced in symmetric e + e~ collisions at the 
Cornell Electron Storage Ring (CESR) and collected with the CLEO detector in two con- 
figurations, known as CLEO II [7] and CLEO II. V [8]. It comprises 9.12 fb" 1 of integrated 
luminosity collected on the T(4S') resonance, corresponding to 9.7 x 10 6 BB pairs, of which 
6.3 x 10 6 were taken with CLEO II. V. An additional 4.36 fb -1 collected below the BB pro- 
duction threshold is used to study non-BB backgrounds. The response of the experimental 
apparatus is studied with a GEANT-based [9] simulation of the CLEO detector, where the 
simulated events are processed in a fashion similar to data. 

In CLEO II, the momenta of charged particles are measured with a tracking system 
consisting of a six-layer straw tube chamber, a ten-layer precision drift chamber, and a 
51-layer main drift chamber, all operating inside a 1.5 T superconducting solenoid. The 
main drift chamber also provides a measurement of specific ionization energy loss (dE/dx), 
which is used for particle identification. For CLEO II. V, the six-layer straw tube chamber 
was replaced by a three-layer double-sided silicon vertex detector, and the gas in the main 
drift chamber was changed from an argon-ethane to a helium-propane mixture. Photons 
are detected with a 7800-crystal Csl electromagnetic calorimeter, which is also inside the 
solenoid. Proportional chambers placed at various depths within the steel return yoke of the 
magnet identify muons. 

Charged tracks are required to be well-measured and to satisfy criteria based on the track 
fit quality and must be consistent with coming from the interaction point in three dimensions. 
Pions and kaons are identified by dE/dx, and tracks that are positively identified as electrons 
or muons are not allowed to form the B candidate. We form ir° candidates from pairs of 
photons with an invariant mass within 20 MeV/c 2 or approximately 2.5 standard deviations 
(er) of the known n° mass. These are then kinematically fitted with the mass constrained 
to the known 7r° mass. We also require the ir° momentum to be greater than 1 GeV/c 
to reduce combinatoric background from low-momentum 7r° candidates. Kg candidates are 
selected from pairs of tracks with invariant mass within 10 MeV/c 2 or 2.5<x of the known K s 
mass. In addition, K s candidates are required to originate from the beam spot and to have 
well-measured displaced decay vertices. 
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We identify B meson candidates by their invariant mass and the total energy of their 
decay products. We calculate a beam-constrained mass by substituting the beam energy 
(Eb) for the measured B candidate energy: M = \J E% — p 2 B , where is the B candidate 
momentum. Performing this substitution improves the resolution of M by one order of 
magnitude, to about 3 MeV/c 2 . We define AE = E\ + E 2 + E 3 — E b , where Ei, E 2 , and E 3 
are the energies of the B candidate daughters. For final states with a Kg and two charged 
tracks, the AE resolution is about 20 MeV for CLEO II and 15 MeV for CLEO II.V. A tt° 
in the final state degrades this resolution by approximately a factor of two. AE is always 
calculated assuming the is a pion. Therefore, the AE distribution for pions is centered 
at zero, while that for kaons is shifted by at least —40 MeV. We accept B candidates with 
M between 5.2 and 5.3 GeV/c 2 and with \AE\ less than 300 MeV for modes containing a it 
and 200 MeV for K s h + 7r~ . This region includes the signal region and a generous sideband 
for background normalization. We reject candidates that are consistent with the exclusive 
b — > c transitions B — > Dn, where D — > Kn, and B — > ipK°, where ip — > and the 

muons are misidentified as pions. 

The main background in this analysis arises from e + e~ — > qq, where q — u,d, s, c. To 
suppress this background, we calculate the angle # sp h between the sphericity axis [10] of 
the tracks and showers forming the B candidate and that of the remainder of the event. 
Because of their two-jet structure, continuum qq events peak strongly at | cos# sp h| = 1, while 
the more isotropic BB events are nearly flat in this variable. By requiring | cos# sp h| < 0.8, 
we reject 83% of the continuum background while retaining roughly the same fraction of 
signal B decays. Additional separation of signal from qq background is provided by a Fisher 
discriminant [11] T formed from eleven variables: the angle between the sphericity axis of the 
candidate and the beam axis, the ratio of Fox- Wolfram moments H 2 /H [12], and the scalar 
sum of the visible momentum in nine 10° angular bins around the candidate sphericity axis. 
We also make use of the angle between the B candidate momentum and the beam axis, 9 b- 
Angular momentum conservation causes B mesons produced through the T(4<S) to exhibit 
a sin 2 9b dependence, while candidates from continuum are flat in cos 9b- 

Our loose selection criteria result in samples consisting primarily of background events 
and containing 11893 candidates for K s h + TT~, 28589 for K + h-n°, and 9339 for K° s h + n°. To 
extract signal yields, we perform unbinned maximum likelihood fits using the observables 
M, AE, T , cos 9b, and the dE/dx of the faster of the two primary tracks (typically with 
momentum above 1 GeV/c) in Kgh + n~ and K + h^n° and of the only primary track in 
Kgh + n°. At high momentum, charged pions and kaons are statistically separated by their 
dE/dx and by AE, each of which provides discrimination at the 2.0<r level (1.7a for CLEO 
II), and we fit for both particle hypotheses simultaneously. Charged pions and kaons with 
momentum below 1 GeV/c are cleanly identified by dE/dx consistency at the 3a level. 

We perform one fit for each topology, Kgh + 7r~, K + h~7T°, and Kgh + n°, allowing for six 
signal and background components, pion and kaon hypotheses for for each of the following: 
signal, continuum background, and background from b — > c decays. The probability for an 
event to be consistent with a given component is the product of the probability distribution 
function (PDF) values for each of the input variables (neglecting correlations). The likelihood 
for each event is the sum of probabilities over the six components, with relative weights 
determined by maximizing the total likelihood of the sample, which is given by the following 
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expression: C = UtT** [j2*= 1 mP ° nCatS (fj Utf^ V ijk )] . The V ijk are the per event PDF 
values, and the fj are the free parameters optimized by the fit and constrained to sum to 
unity. Since the PDFs are normalized to unit integral over the fiducial region, the fj can be 
interpreted as component fractions. The parameters of the dE/dx PDFs are measured from 
D — > K + n~ decays in data. For all other variables, the signal and b — > c PDFs are determined 
from high-statistics Monte Carlo samples, and the continuum PDFs are determined from 
data collected below the BB production threshold. The impact of correlations among the 
input variables is reduced by determining the PDFs as a function of the event location in the 
Dalitz plot, for coarse bins in the M 2 (Kn)-M 2 (nn) plane. We use Monte Carlo simulation 
to estimate the systematic error associated with neglecting any remaining correlations. 

Detection efficiencies and crossfeed among the signal modes are measured from Monte 
Carlo simulated events. The statistical significance of the raw yield N reported by the fit 
is determined by repeating the fit with N fixed to be zero. A significance of no results in 
an increase in — 21n£ of n 2 . We calculate fit yield upper limits (N UL ) at the 90% confi- 

( nUL C N dN 

dence level by integrating the likelihood function: Jo r ^ = 0.90, where £ max (N) is 

the maximum likelihood at fixed N, which conservatively accounts for possible correlations 
among the free parameters in the fit. 

Table I lists the results of the fits to the three topologies. We observe a signal for 
B — > Kgir + ir~ with a statistical significance of 8.1a. Since the efficiency depends on position 
in the Dalitz plot, we evaluate the efficiency in bins across the Dalitz plot and apply a 
correction to each event. The branching fractions and upper limits thus obtained are free 
from model dependence. The efficiencies given in Table I are averages over the efficiencies 
for the observed events weighted by the probability that they are signal. For modes with 
yields consistent with zero, i.e. the three KKn modes, the signal Dalitz plot distribution 
is unknown. Therefore, we examine several models of signal distribution across the Dalitz 
plot, both resonant and non-resonant, and we adopt the model with the lowest measured 
efficiency to establish conservative upper limits. The two sets of errors correspond to the 
statistical and systematic uncertainties, respectively. The largest contributions to the latter 
are uncertainties in the PDF parameters and reconstruction efficiencies. Branching fraction 
upper limits are increased by one standard deviation of the relative systematic uncertainty. 
We assume equal branching fractions for T(AS) — > B°B° and B + B~, which is consistent 
with a recent measurement [13]. 

In these six-component fits, we measure yields for the three-body final states ignoring 
any possible resonant substructure. We also perform fits for each topology that include 
the Dalitz plot variables as inputs and allow for various intermediate resonances (if* (892), 
Kq(1A30), p(770), and / (980)) as well as non-resonant phase space decay. The Dalitz 
plot PDFs include our knowledge of the helicity structure in these decays. We neglect 
interference among these processes and assign a systematic error estimated from Monte Carlo 
simulation. The decays B -> K + (X)h~, where K+(X) denotes if*+(892) or if *+(1430), are 
accessible through different K + (X) submodes in both the K^h + n~ and K + h~ir° topologies. 
To maximize our sensitivity to these decays, we perform the Dalitz plot fits for these two 
topologies simultaneously, with the branching fraction for each K + (X)h~ decay constrained 
to be equal in its two K + (X) submodes, which are related by isospin. 

We perform Dalitz plot fits for the three topologies with differing combinations of in- 
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termediate resonant and non-resonant states, with up to nine signal components. The 
only channel where we observe a statistically significant signal is i3 — > fT* + (892)7r~ with 
a yield of 12.6l£jj for fT*+(892) -> X°7r + and 6.1±?;| for X* + (892) -> fT+vr and a com- 
bined significance of 4. 6c. The size of the signal is insensitive to the choice of other 
resonances included in the fit. In the K s h + -n^ topology, we find the fractional yield 
iV( J FP + (892)7r-)/Ar(K°7r+7r-) to be 0.20±g;g|, which is 7.4a from unity. With efficiencies 
of 8.1% and 3.9% for fT* + (892) — > K°-k + and K + tt°, respectively, we obtain a branching 
fraction of B(B -> fT* + (892)7r-) = (16±% ± 2) x 10~ 6 . Most theoretical predictions [3] for 
this branching fraction lie in the range (2-14) x 10~ 6 . 

Figure 1 shows the event distributions in M and AE for modes where we claim an 
observation. The background in these plots has been suppressed with cuts on the ratios 
of signal and background likelihoods computed without M and AE respectively. Overlaid 
are the fit projections for the signal and background components, scaled by the efficiency 
of the likelihood ratio requirements (40-50% for K°it + it- and 70-80% for K* + (892)n-). 
Figure 2a and 2b show the M(K°ir + ) and M(it + it~) distributions for events in the K s h + -n~ 
fit satisfying a likelihood ratio requirement. Overlaid are the fit predictions for background, 
B — > A'* + (892)7r~, and all other signal modes combined. We define the helicity angle for 
B -> K* + (892)(K°7i + )ir-, 9 heh to be the angle between the K* + (892) daughter tt+ direction 
in the K* + (892) rest frame and the i^* + (892) direction in the B rest frame. Figure 2c shows 
the distribution of cos^hci in the region 0.75 < M(K°n + ) < 1.05 GeV/c 2 after subtracting 
all contributions except B — > fT* + (892)7r~, which are estimated from data collected below 
the BB production threshold and from Monte Carlo simulation. The data are consistent 
with the cos 2 6>h c i dependence expected for a pseudoscalar-vector B decay and reflected in 
the overlaid fit projection. 

In summary, we have observed the three-body decay B — > K Q it + it~ with a branching 
fraction (50tg° ± 7) x 10 -6 . A simultaneous analysis of the Dalitz plots for this decay and 
for B — > K + 7t~7t° reveals the presence of B — * K* + (892)tt~ with a branching fraction 
(16^5 ± 2) x 10~ 6 , which is larger than but consistent with most theoretical predictions. 

TABLE I. Maximum likelihood fit results for three-body decays. Reconstruction efficiencies 
include all daughter branching fractions. The errors on branching fractions B are statistical and 
systematic. Upper limits are computed at the 90% confidence level. 
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FIG. 1. M and AE projections for B -> K°n + ir~ (a and c) and B -» i^*+(892)vr- (b 

and d), which include the two K* + (892) submodes, K*+(892) -> K°vr + (light shade) and 

X* + (892) — » i^ + 7r° (dark shade). The background has been suppressed with cuts on the ratios of 

signal and background likelihoods computed without the displayed variable. The dashed and solid 

lines show the fit predictions for background and the sum of signal and background, respectively. 
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FIG. 2. M(K°7r + ) and M(7r + 7r~) projections for B — > ET°7r + 7r~ (a and b, respectively) and 
the cos^hei projection for B —>■ K* + (892)(K tt + )7t~ (c). Shown in (a) and (b) are the distribution 
for data (points) and the fit predictions for background (dark shade), the sum of all signal B 
decays except B — ► K* + (892)tt~ (medium shade), and B — > K* + (892)7r _ (light shade). Shown 
in (c) are the data distribution (points) in the region 0.75 < M(K°ir + ) < 1.05 GeV/c 2 with all 
non-i^* + (892)7r~ contributions subtracted and the fit prediction for B — > K* + (892)ir~ (histogram). 
The background has been suppressed with a cut on the ratio of signal and background likelihoods 
computed without the Dalitz plot variables. 
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